Well I managed to get a version of the hack running with custom music and 1 custom character. But the character select icon is glitchy, the game crashes if you stay on the character select screen for more than a second, and the game crashes if you choose the custom character. But the character does appear as a CPU normally in Grand Prix mode, so I think I'm getting somewhere.
Is there an archive where I can download older versions of Pitstop? I've tried this on 0.0.5.2, 0.0.5.1, and 0.0.5.0 and I get the same results on all of them when I try custom music with custom characters. The hacks that I've seen with both working were clearly made before 5.0. So I believe that whatever changes were made in recent versions to fix the game crashing at the Battle/VS mode results screen somehow broke compatibility between custom characters and custom music.